Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.Net Agents - Add Streaming support for OpenAIAssistantAgent and AgentChat #8175

Merged
merged 300 commits into from
Sep 9, 2024

Conversation

crickman
Copy link
Contributor

@crickman crickman commented Aug 14, 2024

Motivation and Context

Add streaming support to direct invocation of OpenAIAssistantAgent and also AgentChat:

Fixes: #4833
Fixes: #5643

Integrates with existing streaming support for ChatCompletionAgent.

Description

Streaming returns the expected streaming content types as well as providing whole messages in the provided history.

Support both agent invocation (no-chat) and via AgentChat.

Introduced:

  • StreamingAnnotationContent
  • StreamingFileReferenceContent

Added samples and tests.

Contribution Checklist

crickman and others added 30 commits July 7, 2024 14:49
…#7130)

### Motivation and Context
This PR migrates the `AzureOpenAIAudioToTextService` to Azure.AI.OpenAI
SDK v2.

### Description
1. The existing `OpenAIAudioToTextExecutionSettings` class is copied to
the new `Connectors.AzureOpenAI` project and renamed to
`AzureOpenAIAudioToTextExecutionSettings` to represent prompt execution
settings for the `AzureOpenAIAudioToTextService`.
2. The `OpenAIAudioToTextExecutionSettings.ResponseFormat` property type
has changed from string to the `AudioTranscriptionFormat` enum, which is
a breaking change that is tracked in the issue -
https://github.com/microsoft/semantic-kernel/issues/70533.
3. The `ClientCore.AudioToText.cs` class is refactored to use the new
`AzureOpenAIAudioToTextExecutionSettings` class and to handle the new
type of `OpenAIAudioToTextExecutionSettings.ResponseFormat` property.
4. Service collection and kernel builder extension methods are added to
register the service in the DI container.
5. Unit and integration tests are added as well.
…for format (#7141)

### Motivation and Context

- Updates Execution Settings for `ResponseFormat` from `string` to
`enum`.
- Updates the Unit Tests to validate.
### Motivation, Context, and Description
1. The `ClientCore.DeploymentOrModelName` property is renamed to
`ClientCore.DeploymentName` to reflect it's actual purpose.
2. The `AzureOpenAIPromptExecutionSettings` class is moved to the
`Settings` folder to reside alongside prompt execution setting classes
for other services.
…7148)

### Motivation, Context and Description
This PR:
1. Copies OpenAIFileService and the other classes related to it to the
Connectors.AzureOpenAI project before refactoring them to use the
Azure.AI.OpenAI SDK v2.
2. Renames the classes and changes namespaces accordingly.  
3. Adds/changes no functionality in any of those classes.  
4. Temporarily removes the classes from the compilation process. They
will be included back in the next PR.

---------

Co-authored-by: Roger Barreto <[email protected]>
### Motivation and Context

This PR does the following:  
1. Migrates OpenAIChatCompletionService, ClientCore, and other model
classes both use, to OpenAI SDK v2.
2. Updates ToolCallBehavior classes to return a list of functions and
function choice. This change is required because the new SDK model
requires both of those for the CompletionsOptions class creation and
does not allow setting them after the class is already created, as it
used to allow.
3. Adapts related unit tests to the API changes.
4. Added netstandard2.0 as target for the UnitTest project.
5. Rename `TryGetFunctionAndArguments` to
`TryGetOpenAIFunctionAndArguments` to avoid clashing with AzureOpenAI
### Motivation and Context

- Obsolescence of FileService.
…#7203)

### Motivation and Context

As a way to enforce the Non Breaking principle this PR remove all
avoidable breaking changes from the V2 migration, including additions
and small improvements identified.
### Motivation and Context
It was decided not to migrate existing `OpenAIFileService` to the
Azure.AI.OpenAI SDK v2 and eventually deprecate it.

### Description
This PR removes files related to the `OpenAIFileService` added by a
previous PR.
### Motivation and Context

Removing non existing V1 logging.
### Motivation and Context
This PR aligns metadata names with those provided by the underlying
Azure.AI.OpenAI library. Additionally, it adds a few unit tests to
increase code coverage.
### Motivation, Context and Description
The new Azure.AI.OpenAI V2 introduced timestamp granularities for the
audio-to-text API. This PR removes the first attempt to expose the
settings in execution settings for the service. The setting will be
introduced later after the migration is over in scope in this task:
#7213.
### Motivation, Context and Description
The unnecessary breaking change that was introduced earlier, which
changed the type of the
`OpenAIAudioToTextExecutionSettings.ResponseFormat` and
`AzureOpenAIAudioToTextExecutionSettings.ResponseFormat` properties, has
been rolled back.
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 6, 2024
@crickman crickman added this pull request to the merge queue Sep 6, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 6, 2024
@crickman crickman added this pull request to the merge queue Sep 6, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 6, 2024
@crickman crickman added this pull request to the merge queue Sep 6, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 6, 2024
@crickman crickman added this pull request to the merge queue Sep 6, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 6, 2024
@crickman crickman added this pull request to the merge queue Sep 7, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 7, 2024
@crickman crickman added this pull request to the merge queue Sep 9, 2024
@crickman crickman removed this pull request from the merge queue due to a manual request Sep 9, 2024
@crickman crickman added this pull request to the merge queue Sep 9, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 9, 2024
@crickman crickman added this pull request to the merge queue Sep 9, 2024
@crickman crickman removed this pull request from the merge queue due to a manual request Sep 9, 2024
@crickman crickman added this pull request to the merge queue Sep 9, 2024
github-merge-queue bot pushed a commit that referenced this pull request Sep 9, 2024
…gentChat` (#8175)

### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

Add streaming support to direct invocation of `OpenAIAssistantAgent` and
also `AgentChat`:

Fixes: #4833
Fixes: #5643

Integrates with existing streaming support for `ChatCompletionAgent`.

### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

Streaming returns the expected streaming content types as well as
providing whole messages in the provided history.

Support both agent invocation (no-chat) and via `AgentChat`.

Introduced:
- `StreamingAnnotationContent`
- `StreamingFileReferenceContent`

Added samples and tests.

### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄

---------

Co-authored-by: SergeyMenshykh <[email protected]>
Co-authored-by: Roger Barreto <[email protected]>
Co-authored-by: SergeyMenshykh <[email protected]>
Co-authored-by: Roger Barreto <[email protected]>
Co-authored-by: Dmytro Struk <[email protected]>
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 9, 2024
@crickman crickman added this pull request to the merge queue Sep 9, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 9, 2024
@crickman crickman added this pull request to the merge queue Sep 9, 2024
Merged via the queue into main with commit 6426911 Sep 9, 2024
18 checks passed
@crickman crickman deleted the feature_agents_streaming branch September 9, 2024 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
agents enhancement experimental Associated with an experimental feature Ignite kernel.core kernel Issues or pull requests impacting the core kernel .NET Issue or Pull requests regarding .NET code PR: ready for review All feedback addressed, ready for reviews
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

.Net Agents: Assistants V2 - Support for streaming .Net Agents: Add streaming to agent framework
7 participants